SherbyNet - /page/wikibook/entry.mu

<!DOCTYPE html>

<title>Matrix regularization</title>

</head>

Matrix regularization

</h1>

</header>

<div id="mw-content-text" class="mw-body-content mw-content-ltr" lang="en" dir="ltr"><div class="mw-content-ltr mw-parser-output" lang="en" dir="ltr">In the field of <a href="Statistical_learning_theory" title="Statistical learning theory">statistical learning theory</a>, matrix regularization generalizes notions of vector regularization to cases where the object to be learned is a matrix. The purpose of regularization is to enforce conditions, for example sparsity or smoothness, that can produce stable predictive functions. For example, in the more common vector framework, <a href="Tikhonov_regularization" class="mw-redirect" title="Tikhonov regularization">Tikhonov regularization</a> optimizes over

</mrow>

</munder>

<msup>

<mrow>

<mrow>

</mrow>

</mrow>

</mrow>

</msup>

<msup>

<mrow>

</mrow>

</mrow>

</msup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \min _{x}\left\|Ax-y\right\|^{2}+\lambda \left\|x\right\|^{2}}</annotation>

</semantics>

</math>

to find a vector <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle x}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle x}</annotation>

</semantics>

</math><img src="./87f9e315fd7e2ba406057a97300593c4802b53e4.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.33ex; height:1.676ex;" alt="{\displaystyle x}" loading="lazy"> that is a stable solution to the regression problem. When the system is described by a matrix rather than a vector, this problem can be written as

</mrow>

</munder>

<msup>

<mrow>

<mrow>

</mrow>

</mrow>

</mrow>

</msup>

<msup>

<mrow>

</mrow>

</mrow>

</msup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \min _{X}\left\|AX-Y\right\|^{2}+\lambda \left\|X\right\|^{2},}</annotation>

</semantics>

</math>

where the vector norm enforcing a regularization penalty on <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle x}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle x}</annotation>

</semantics>

</math><img src="./87f9e315fd7e2ba406057a97300593c4802b53e4.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.33ex; height:1.676ex;" alt="{\displaystyle x}" loading="lazy"> has been extended to a matrix norm on <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle X}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle X}</annotation>

</semantics>

</math><img src="./68baa052181f707c662844a465bfeeb135e82bab.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.98ex; height:2.176ex;" alt="{\displaystyle X}" loading="lazy">.

Matrix regularization has applications in <a href="Matrix_completion" title="Matrix completion">matrix completion</a>, <a href="Multivariate_regression" class="mw-redirect" title="Multivariate regression">multivariate regression</a>, and <a href="Multi-task_learning" title="Multi-task learning">multi-task learning</a>. Ideas of feature and group selection can also be extended to matrices, and these can be generalized to the nonparametric case of <a href="Multiple_kernel_learning" title="Multiple kernel learning">multiple kernel learning</a>.

<div class="mw-heading mw-heading2"><h2 id="Basic_definition">Basic definition</h2></div>

Consider a matrix <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle W}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle W}</annotation>

</semantics>

</math><img src="./54a9c4c547f4d6111f81946cad242b18298d70b7.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:2.435ex; height:2.176ex;" alt="{\displaystyle W}" loading="lazy"> to be learned from a set of examples, <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle S=(X_{i}^{t},y_{i}^{t})}">

</mrow>

</mrow>

</msubsup>

</mrow>

</mrow>

</msubsup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle S=(X_{i}^{t},y_{i}^{t})}</annotation>

</semantics>

</math><img src="./17c9d072460edc44cef87f0aa802f21994d51091.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.005ex; width:12.25ex; height:3.176ex;" alt="{\displaystyle S=(X_{i}^{t},y_{i}^{t})}" loading="lazy">, where <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle i}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle i}</annotation>

</semantics>

</math><img src="./add78d8608ad86e54951b8c8bd6c8d8416533d20.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:0.802ex; height:2.176ex;" alt="{\displaystyle i}" loading="lazy"> goes from <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle 1}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle 1}</annotation>

</semantics>

</math><img src="./92d98b82a3778f043108d4e20960a9193df57cbf.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.162ex; height:2.176ex;" alt="{\displaystyle 1}" loading="lazy"> to <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle n}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle n}</annotation>

</semantics>

</math><img src="./a601995d55609f2d9f5e233e36fbe9ea26011b3b.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.395ex; height:1.676ex;" alt="{\displaystyle n}" loading="lazy">, and <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle t}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle t}</annotation>

</semantics>

</math><img src="./65658b7b223af9e1acc877d848888ecdb4466560.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:0.84ex; height:2.009ex;" alt="{\displaystyle t}" loading="lazy"> goes from <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle 1}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle 1}</annotation>

</semantics>

</math><img src="./92d98b82a3778f043108d4e20960a9193df57cbf.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.162ex; height:2.176ex;" alt="{\displaystyle 1}" loading="lazy"> to <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle T}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle T}</annotation>

</semantics>

</math><img src="./ec7200acd984a1d3a3d7dc455e262fbe54f7f6e0.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.636ex; height:2.176ex;" alt="{\displaystyle T}" loading="lazy">. Let each input matrix <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle X_{i}}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle X_{i}}</annotation>

</semantics>

</math><img src="./af4a0955af42beb5f85aa05fb8c07abedc13990d.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:2.724ex; height:2.509ex;" alt="{\displaystyle X_{i}}" loading="lazy"> be <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \in \mathbb {R} ^{DT}}">

<msup>

</mrow>

</mrow>

</msup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \in \mathbb {R} ^{DT}}</annotation>

</semantics>

</math><img src="./81ba510648d2a0a8b3d776c5c05d64805c1680ee.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:6.623ex; height:2.676ex;" alt="{\displaystyle \in \mathbb {R} ^{DT}}" loading="lazy">, and let <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle W}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle W}</annotation>

</semantics>

</math><img src="./54a9c4c547f4d6111f81946cad242b18298d70b7.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:2.435ex; height:2.176ex;" alt="{\displaystyle W}" loading="lazy"> be of size <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle D\times T}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle D\times T}</annotation>

</semantics>

</math><img src="./c427ea48a6ded5f70b20137be4648fcbd41991d3.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:6.401ex; height:2.176ex;" alt="{\displaystyle D\times T}" loading="lazy">. A general model for the output <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle y}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle y}</annotation>

</semantics>

</math><img src="./b8a6208ec717213d4317e666f1ae872e00620a0d.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:1.155ex; height:2.009ex;" alt="{\displaystyle y}" loading="lazy"> can be posed as

</mrow>

</mrow>

</msubsup>

<msub>

<mrow>

<mrow>

</mrow>

</mrow>

</msubsup>

</mrow>

</mrow>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle y_{i}^{t}=\left\langle W,X_{i}^{t}\right\rangle _{F},}</annotation>

</semantics>

</math>

where the inner product is the <a href="Frobenius_inner_product" title="Frobenius inner product">Frobenius inner product</a>. For different applications the matrices <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle X_{i}}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle X_{i}}</annotation>

</semantics>

</math><img src="./af4a0955af42beb5f85aa05fb8c07abedc13990d.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:2.724ex; height:2.509ex;" alt="{\displaystyle X_{i}}" loading="lazy"> will have different forms,<a href="#cite_note-Notes-1">[1]</a> but for each of these the optimization problem to infer <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle W}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle W}</annotation>

</semantics>

</math><img src="./54a9c4c547f4d6111f81946cad242b18298d70b7.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:2.435ex; height:2.176ex;" alt="{\displaystyle W}" loading="lazy"> can be written as

</mrow>

</munder>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \min _{W\in {\mathcal {H}}}E(W)+R(W),}</annotation>

</semantics>

</math>

where <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle E}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle E}</annotation>

</semantics>

</math><img src="./4232c9de2ee3eec0a9c0a19b15ab92daa6223f9b.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.776ex; height:2.176ex;" alt="{\displaystyle E}" loading="lazy"> defines the empirical error for a given <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle W}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle W}</annotation>

</semantics>

</math><img src="./54a9c4c547f4d6111f81946cad242b18298d70b7.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:2.435ex; height:2.176ex;" alt="{\displaystyle W}" loading="lazy">, and <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle R(W)}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle R(W)}</annotation>

</semantics>

</math><img src="./7659904dc54c8b8ba1edb8f25e6407a345610053.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:6.009ex; height:2.843ex;" alt="{\displaystyle R(W)}" loading="lazy"> is a matrix regularization penalty. The function <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle R(W)}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle R(W)}</annotation>

</semantics>

</math><img src="./7659904dc54c8b8ba1edb8f25e6407a345610053.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:6.009ex; height:2.843ex;" alt="{\displaystyle R(W)}" loading="lazy"> is typically chosen to be convex and is often selected to enforce sparsity (using <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \ell ^{1}}">

<msup>

</mrow>

</msup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \ell ^{1}}</annotation>

</semantics>

</math><img src="./7d1156e1c2220628042b0fc51e0c73deb3b7c6d1.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:2.024ex; height:2.676ex;" alt="{\displaystyle \ell ^{1}}" loading="lazy">-norms) and/or smoothness (using <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \ell ^{2}}">

<msup>

</mrow>

</msup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \ell ^{2}}</annotation>

</semantics>

</math><img src="./91f1f909abd70bd3d8fff0f7ae1ac23052387e18.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:2.024ex; height:2.676ex;" alt="{\displaystyle \ell ^{2}}" loading="lazy">-norms). Finally, <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle W}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle W}</annotation>

</semantics>

</math><img src="./54a9c4c547f4d6111f81946cad242b18298d70b7.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:2.435ex; height:2.176ex;" alt="{\displaystyle W}" loading="lazy"> is in the space of matrices <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle {\mathcal {H}}}">

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\mathcal {H}}}</annotation>

</semantics>

</math><img src="./19ef4c7b923a5125ac91aa491838a95ee15b804f.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.964ex; height:2.176ex;" alt="{\displaystyle {\mathcal {H}}}" loading="lazy"> with Frobenius inner product <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \langle \dots \rangle _{F}}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \langle \dots \rangle _{F}}</annotation>

</semantics>

</math><img src="./8629828e1dfe1ae5d9fb58995bea5e341d4e2333.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:5.996ex; height:2.843ex;" alt="{\displaystyle \langle \dots \rangle _{F}}" loading="lazy">.

<div class="mw-heading mw-heading2"><h2 id="General_applications">General applications</h2></div>

<div class="mw-heading mw-heading3"><h3 id="Matrix_completion">Matrix completion</h3></div>

In the problem of <a href="Matrix_completion" title="Matrix completion">matrix completion</a>, the matrix <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle X_{i}^{t}}">

</mrow>

</mrow>

</msubsup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle X_{i}^{t}}</annotation>

</semantics>

</math><img src="./ee1e4707e18dd771673dcc15524a2ad70218b1a5.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.005ex; width:2.823ex; height:3.176ex;" alt="{\displaystyle X_{i}^{t}}" loading="lazy"> takes the form

</mrow>

</mrow>

</msubsup>

<msub>

</mrow>

</msub>

</mrow>

</msubsup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle X_{i}^{t}=e_{t}\otimes e_{i}',}</annotation>

</semantics>

</math>

where <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle (e_{t})_{t}}">

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle (e_{t})_{t}}</annotation>

</semantics>

</math><img src="./17d17d057582ec4972f9b91b76b9211b8bb31e01.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:4.545ex; height:2.843ex;" alt="{\displaystyle (e_{t})_{t}}" loading="lazy"> and <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle (e_{i}')_{i}}">

</mrow>

</msubsup>

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle (e_{i}')_{i}}</annotation>

</semantics>

</math><img src="./ed5c8e09bc2c11c42e49ca5d0b0d4a904ab8267b.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.005ex; width:4.492ex; height:3.009ex;" alt="{\displaystyle (e_{i}')_{i}}" loading="lazy"> are the canonical basis in <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \mathbb {R} ^{T}}">

<msup>

</mrow>

</mrow>

</msup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \mathbb {R} ^{T}}</annotation>

</semantics>

</math><img src="./a1bed70312af4ab7ee5eed503cd5c54575ca513b.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:3.067ex; height:2.676ex;" alt="{\displaystyle \mathbb {R} ^{T}}" loading="lazy"> and <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \mathbb {R} ^{D}}">

<msup>

</mrow>

</mrow>

</msup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \mathbb {R} ^{D}}</annotation>

</semantics>

</math><img src="./98be01474d92584e89724b486a4d85914533c29e.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:3.271ex; height:2.676ex;" alt="{\displaystyle \mathbb {R} ^{D}}" loading="lazy">. In this case the role of the Frobenius inner product is to select individual elements <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle w_{i}^{t}}">

</mrow>

</mrow>

</msubsup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle w_{i}^{t}}</annotation>

</semantics>

</math><img src="./633bd9e54ef112edddb35243b1363b2a13b29267.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.005ex; width:2.49ex; height:3.176ex;" alt="{\displaystyle w_{i}^{t}}" loading="lazy"> from the matrix <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle W}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle W}</annotation>

</semantics>

</math><img src="./54a9c4c547f4d6111f81946cad242b18298d70b7.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:2.435ex; height:2.176ex;" alt="{\displaystyle W}" loading="lazy">. Thus, the output <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle y}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle y}</annotation>

</semantics>

</math><img src="./b8a6208ec717213d4317e666f1ae872e00620a0d.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:1.155ex; height:2.009ex;" alt="{\displaystyle y}" loading="lazy"> is a sampling of entries from the matrix <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle W}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle W}</annotation>

</semantics>

</math><img src="./54a9c4c547f4d6111f81946cad242b18298d70b7.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:2.435ex; height:2.176ex;" alt="{\displaystyle W}" loading="lazy">.

The problem of reconstructing <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle W}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle W}</annotation>

</semantics>

</math><img src="./54a9c4c547f4d6111f81946cad242b18298d70b7.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:2.435ex; height:2.176ex;" alt="{\displaystyle W}" loading="lazy"> from a small set of sampled entries is possible only under certain restrictions on the matrix, and these restrictions can be enforced by a regularization function. For example, it might be assumed that <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle W}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle W}</annotation>

</semantics>

</math><img src="./54a9c4c547f4d6111f81946cad242b18298d70b7.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:2.435ex; height:2.176ex;" alt="{\displaystyle W}" loading="lazy"> is low-rank, in which case the regularization penalty can take the form of a nuclear norm.<a href="#cite_note-Candès,_Emmanuel_J_2009_pp._717-2">[2]</a>

<msub>

<mrow>

</mrow>

</mrow>

</msub>

</mrow>

</munder>

<mrow>

<msub>

</mrow>

</msub>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle R(W)=\lambda \left\|W\right\|_{*}=\lambda \sum _{i}\left|\sigma _{i}\right|,}</annotation>

</semantics>

</math>

where <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \sigma _{i}}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \sigma _{i}}</annotation>

</semantics>

</math><img src="./6ab3208a7d0c634ef720e03ff5a9949e8310edc4.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:2.127ex; height:2.009ex;" alt="{\displaystyle \sigma _{i}}" loading="lazy">, with <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle i}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle i}</annotation>

</semantics>

</math><img src="./add78d8608ad86e54951b8c8bd6c8d8416533d20.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:0.802ex; height:2.176ex;" alt="{\displaystyle i}" loading="lazy"> from <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle 1}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle 1}</annotation>

</semantics>

</math><img src="./92d98b82a3778f043108d4e20960a9193df57cbf.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.162ex; height:2.176ex;" alt="{\displaystyle 1}" loading="lazy"> to <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \min D,T}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \min D,T}</annotation>

</semantics>

</math><img src="./60869621f725308ac600ea281576ac6a34fd6c21.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:8.857ex; height:2.509ex;" alt="{\displaystyle \min D,T}" loading="lazy">, are the singular values of <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle W}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle W}</annotation>

</semantics>

</math><img src="./54a9c4c547f4d6111f81946cad242b18298d70b7.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:2.435ex; height:2.176ex;" alt="{\displaystyle W}" loading="lazy">.

<div class="mw-heading mw-heading3"><h3 id="Multivariate_regression">Multivariate regression</h3></div>

Models used in <a href="Multivariate_regression" class="mw-redirect" title="Multivariate regression">multivariate regression</a> are parameterized by a matrix of coefficients. In the Frobenius inner product above, each matrix <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle X}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle X}</annotation>

</semantics>

</math><img src="./68baa052181f707c662844a465bfeeb135e82bab.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.98ex; height:2.176ex;" alt="{\displaystyle X}" loading="lazy"> is

</mrow>

</mrow>

</msubsup>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle X_{i}^{t}=e_{t}\otimes x_{i}}</annotation>

</semantics>

</math>

such that the output of the inner product is the <a href="Dot_product" title="Dot product">dot product</a> of one row of the input with one column of the coefficient matrix. The familiar form of such models is

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle Y=XW+b}</annotation>

</semantics>

</math>

Many of the vector norms used in single variable regression can be extended to the multivariate case. One example is the squared Frobenius norm, which can be viewed as an <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \ell ^{2}}">

<msup>

</mrow>

</msup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \ell ^{2}}</annotation>

</semantics>

</math><img src="./91f1f909abd70bd3d8fff0f7ae1ac23052387e18.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:2.024ex; height:2.676ex;" alt="{\displaystyle \ell ^{2}}" loading="lazy">-norm acting either entrywise, or on the singular values of the matrix:

<mrow>

</mrow>

</mrow>

</mrow>

</msubsup>

</mrow>

</munder>

</mrow>

</munder>

<msup>

<mrow>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</msup>

<mrow>

<mrow>

<msup>

</mrow>

</msup>

</mrow>

</mrow>

</mrow>

</munder>

</mrow>

</mrow>

</msubsup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle R(W)=\lambda \left\|W\right\|_{F}^{2}=\lambda \sum _{i}\sum _{j}\left|w_{ij}\right|^{2}=\lambda \operatorname {Tr} \left(W^{*}W\right)=\lambda \sum _{i}\sigma _{i}^{2}.}</annotation>

</semantics>

</math>

In the multivariate case the effect of regularizing with the Frobenius norm is the same as the vector case; very complex models will have larger norms, and, thus, will be penalized more.

<div class="mw-heading mw-heading3"><h3 id="Multi-task_learning">Multi-task learning</h3></div>

The setup for multi-task learning is almost the same as the setup for multivariate regression. The primary difference is that the input variables are also indexed by task (columns of <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle Y}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle Y}</annotation>

</semantics>

</math><img src="./961d67d6b454b4df2301ac571808a3538b3a6d3f.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.171ex; width:1.773ex; height:2.009ex;" alt="{\displaystyle Y}" loading="lazy">). The representation with the Frobenius inner product is then

</mrow>

</mrow>

</msubsup>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</msubsup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle X_{i}^{t}=e_{t}\otimes x_{i}^{t}.}</annotation>

</semantics>

</math>

The role of matrix regularization in this setting can be the same as in multivariate regression, but matrix norms can also be used to couple learning problems across tasks. In particular, note that for the optimization problem

</mrow>

</munder>

<mrow>

<mrow>

</mrow>

</mrow>

</mrow>

</mrow>

</msubsup>

<mrow>

</mrow>

</mrow>

</mrow>

</msubsup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \min _{W}\left\|XW-Y\right\|_{2}^{2}+\lambda \left\|W\right\|_{2}^{2}}</annotation>

</semantics>

</math>

the solutions corresponding to each column of <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle Y}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle Y}</annotation>

</semantics>

</math><img src="./961d67d6b454b4df2301ac571808a3538b3a6d3f.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.171ex; width:1.773ex; height:2.009ex;" alt="{\displaystyle Y}" loading="lazy"> are decoupled. That is, the same solution can be found by solving the joint problem, or by solving an isolated regression problem for each column. The problems can be coupled by adding an additional regularization penalty on the covariance of solutions

</mrow>

</munder>

<mrow>

<mrow>

</mrow>

</mrow>

</mrow>

</mrow>

</msubsup>

<msub>

</mrow>

</msub>

<mrow>

</mrow>

</mrow>

</mrow>

</msubsup>

<msub>

</mrow>

</msub>

<mrow>

<mrow>

<msup>

</mrow>

</msup>

<msup>

</mrow>

</msup>

</mrow>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \min _{W,\Omega }\left\|XW-Y\right\|_{2}^{2}+\lambda _{1}\left\|W\right\|_{2}^{2}+\lambda _{2}\operatorname {Tr} \left(W^{T}\Omega ^{-1}W\right)}</annotation>

</semantics>

</math>

where <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \Omega }">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \Omega }</annotation>

</semantics>

</math><img src="./24b0d5ca6f381068d756f6337c08e0af9d1eeb6f.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.678ex; height:2.176ex;" alt="{\displaystyle \Omega }" loading="lazy"> models the relationship between tasks. This scheme can be used to both enforce similarity of solutions across tasks, and to learn the specific structure of task similarity by alternating between optimizations of <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle W}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle W}</annotation>

</semantics>

</math><img src="./54a9c4c547f4d6111f81946cad242b18298d70b7.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:2.435ex; height:2.176ex;" alt="{\displaystyle W}" loading="lazy"> and <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \Omega }">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \Omega }</annotation>

</semantics>

</math><img src="./24b0d5ca6f381068d756f6337c08e0af9d1eeb6f.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.678ex; height:2.176ex;" alt="{\displaystyle \Omega }" loading="lazy">.<a href="#cite_note-3">[3]</a> When the relationship between tasks is known to lie on a graph, the <a href="Laplacian_matrix" title="Laplacian matrix">Laplacian matrix</a> of the graph can be used to couple the learning problems.

<div class="mw-heading mw-heading2"><h2 id="Spectral_regularization">Spectral regularization</h2></div>

<a href="Regularization_by_spectral_filtering" title="Regularization by spectral filtering">Regularization by spectral filtering</a> has been used to find stable solutions to problems such as those discussed above by addressing ill-posed matrix inversions (see for example <a href="Regularization_by_spectral_filtering#Filter_function_for_Tikhonov_regularization" title="Regularization by spectral filtering">Filter function for Tikhonov regularization</a>). In many cases the regularization function acts on the input (or kernel) to ensure a bounded inverse by eliminating small singular values, but it can also be useful to have spectral norms that act on the matrix that is to be learned.

There are a number of matrix norms that act on the singular values of the matrix. Frequently used examples include the <a href="Schatten_norm" title="Schatten norm">Schatten p-norms</a>, with p = 1 or 2. For example, matrix regularization with a Schatten 1-norm, also called the nuclear norm, can be used to enforce sparsity in the spectrum of a matrix. This has been used in the context of matrix completion when the matrix in question is believed to have a restricted rank.<a href="#cite_note-Candès,_Emmanuel_J_2009_pp._717-2">[2]</a> In this case the optimization problem becomes:

<msub>

<mrow>

</mrow>

</mrow>

</msub>

<mtext> subject to </mtext>

</mrow>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \min \left\|W\right\|_{*}~~{\text{ subject to }}~~W_{i,j}=Y_{ij}.}</annotation>

</semantics>

</math>

Spectral Regularization is also used to enforce a reduced rank coefficient matrix in multivariate regression.<a href="#cite_note-4">[4]</a> In this setting, a reduced rank coefficient matrix can be found by keeping just the top <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle n}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle n}</annotation>

</semantics>

</math><img src="./a601995d55609f2d9f5e233e36fbe9ea26011b3b.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.395ex; height:1.676ex;" alt="{\displaystyle n}" loading="lazy"> singular values, but this can be extended to keep any reduced set of singular values and vectors.

<div class="mw-heading mw-heading2"><h2 id="Structured_sparsity">Structured sparsity</h2></div>

Sparse optimization has become the focus of much research interest as a way to find solutions that depend on a small number of variables (see e.g. the <a href="Lasso_(statistics)" title="Lasso (statistics)">Lasso method</a>). In principle, entry-wise sparsity can be enforced by penalizing the entry-wise <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \ell ^{0}}">

<msup>

</mrow>

</msup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \ell ^{0}}</annotation>

</semantics>

</math><img src="./e822532cdd2bf1925f98fe29820de78213dd5769.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:2.024ex; height:2.676ex;" alt="{\displaystyle \ell ^{0}}" loading="lazy">-norm of the matrix, but the <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \ell ^{0}}">

<msup>

</mrow>

</msup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \ell ^{0}}</annotation>

</semantics>

</math><img src="./e822532cdd2bf1925f98fe29820de78213dd5769.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:2.024ex; height:2.676ex;" alt="{\displaystyle \ell ^{0}}" loading="lazy">-norm is not convex. In practice this can be implemented by convex relaxation to the <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \ell ^{1}}">

<msup>

</mrow>

</msup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \ell ^{1}}</annotation>

</semantics>

</math><img src="./7d1156e1c2220628042b0fc51e0c73deb3b7c6d1.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:2.024ex; height:2.676ex;" alt="{\displaystyle \ell ^{1}}" loading="lazy">-norm. While entry-wise regularization with an <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \ell ^{1}}">

<msup>

</mrow>

</msup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \ell ^{1}}</annotation>

</semantics>

</math><img src="./7d1156e1c2220628042b0fc51e0c73deb3b7c6d1.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:2.024ex; height:2.676ex;" alt="{\displaystyle \ell ^{1}}" loading="lazy">-norm will find solutions with a small number of nonzero elements, applying an <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \ell ^{1}}">

<msup>

</mrow>

</msup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \ell ^{1}}</annotation>

</semantics>

</math><img src="./7d1156e1c2220628042b0fc51e0c73deb3b7c6d1.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:2.024ex; height:2.676ex;" alt="{\displaystyle \ell ^{1}}" loading="lazy">-norm to different groups of variables can enforce structure in the sparsity of solutions.<a href="#cite_note-5">[5]</a>

The most straightforward example of structured sparsity uses the <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \ell _{p,q}}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \ell _{p,q}}</annotation>

</semantics>

</math><img src="./2863b2ca368d1141eacde8c46ac49857c115fd71.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.005ex; width:3.243ex; height:2.843ex;" alt="{\displaystyle \ell _{p,q}}" loading="lazy"> norm with <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle p=2}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle p=2}</annotation>

</semantics>

</math><img src="./d62e4100b94c1939c67f2d4b8580d26c78106c44.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; margin-left: -0.089ex; width:5.52ex; height:2.509ex;" alt="{\displaystyle p=2}" loading="lazy"> and <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle q=1}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle q=1}</annotation>

</semantics>

</math><img src="./785938d022f0b0b0bf4b3afa5e1cedceab7a3874.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:5.33ex; height:2.509ex;" alt="{\displaystyle q=1}" loading="lazy">:

<msub>

<mrow>

</mrow>

</mrow>

</msub>

</mrow>

</munder>

<msub>

<mrow>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \left\|W\right\|_{2,1}=\sum _{i}\left\|w_{i}\right\|_{2}.}</annotation>

</semantics>

</math>

For example, the <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \ell _{2,1}}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \ell _{2,1}}</annotation>

</semantics>

</math><img src="./066787a72536917c808f3d2c364921b4227e50e3.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.005ex; width:3.303ex; height:2.843ex;" alt="{\displaystyle \ell _{2,1}}" loading="lazy"> norm is used in multi-task learning to group features across tasks, such that all the elements in a given row of the coefficient matrix can be forced to zero as a group.<a href="#cite_note-6">[6]</a> The grouping effect is achieved by taking the <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \ell ^{2}}">

<msup>

</mrow>

</msup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \ell ^{2}}</annotation>

</semantics>

</math><img src="./91f1f909abd70bd3d8fff0f7ae1ac23052387e18.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:2.024ex; height:2.676ex;" alt="{\displaystyle \ell ^{2}}" loading="lazy">-norm of each row, and then taking the total penalty to be the sum of these row-wise norms. This regularization results in rows that will tend to be all zeros, or dense. The same type of regularization can be used to enforce sparsity column-wise by taking the <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \ell ^{2}}">

<msup>

</mrow>

</msup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \ell ^{2}}</annotation>

</semantics>

</math><img src="./91f1f909abd70bd3d8fff0f7ae1ac23052387e18.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:2.024ex; height:2.676ex;" alt="{\displaystyle \ell ^{2}}" loading="lazy">-norms of each column.

More generally, the <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \ell _{2,1}}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \ell _{2,1}}</annotation>

</semantics>

</math><img src="./066787a72536917c808f3d2c364921b4227e50e3.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.005ex; width:3.303ex; height:2.843ex;" alt="{\displaystyle \ell _{2,1}}" loading="lazy"> norm can be applied to arbitrary groups of variables:

</mrow>

</mrow>

</munderover>

<msqrt>

</mrow>

</mrow>

<msub>

</mrow>

</msub>

</mrow>

</munderover>

<msup>

<mrow>

</mrow>

</mrow>

</msubsup>

</mrow>

</mrow>

</msup>

</msqrt>

</mrow>

</mrow>

</mrow>

</munderover>

<msub>

<mrow>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle R(W)=\lambda \sum _{g}^{G}{\sqrt {\sum _{j}^{|G_{g}|}\left|w_{g}^{j}\right|^{2}}}=\lambda \sum _{g}^{G}\left\|w_{g}\right\|_{g}}</annotation>

</semantics>

</math>

where the index <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle g}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle g}</annotation>

</semantics>

</math><img src="./d3556280e66fe2c0d0140df20935a6f057381d77.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:1.116ex; height:2.009ex;" alt="{\displaystyle g}" loading="lazy"> is across groups of variables, and <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle |G_{g}|}">

</mrow>

<msub>

</mrow>

</msub>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle |G_{g}|}</annotation>

</semantics>

</math><img src="./2074f3611f52b4df90f65618a731d20d93be0964.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.005ex; width:4.142ex; height:3.009ex;" alt="{\displaystyle |G_{g}|}" loading="lazy"> indicates the cardinality of group <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle g}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle g}</annotation>

</semantics>

</math><img src="./d3556280e66fe2c0d0140df20935a6f057381d77.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:1.116ex; height:2.009ex;" alt="{\displaystyle g}" loading="lazy">.

Algorithms for solving these group sparsity problems extend the more well-known Lasso and group Lasso methods by allowing overlapping groups, for example, and have been implemented via <a href="Matching_pursuit" title="Matching pursuit">matching pursuit</a>:<a href="#cite_note-7">[7]</a> and <a href="Proximal_gradient_method" title="Proximal gradient method">proximal gradient methods</a>.<a href="#cite_note-8">[8]</a> By writing the proximal gradient with respect to a given coefficient, <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle w_{g}^{i}}">

</mrow>

</mrow>

</msubsup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle w_{g}^{i}}</annotation>

</semantics>

</math><img src="./0026ad498d507d33956c5c6b55d7c54299c725b4.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.005ex; width:2.686ex; height:3.176ex;" alt="{\displaystyle w_{g}^{i}}" loading="lazy">, it can be seen that this norm enforces a group-wise soft threshold<a href="#cite_note-Notes-1">[1]</a>

<msub>

<msub>

</mrow>

</msub>

</mrow>

</msub>

<msup>

<mrow>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</msup>

<mrow>

<mrow>

</mrow>

</mrow>

</msubsup>

<mfrac>

</mrow>

</mrow>

</msubsup>

<msub>

<mrow>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</msub>

</mfrac>

</mrow>

</mrow>

<msub>

</mrow>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \operatorname {prox} _{\lambda ,R_{g}}\left(w_{g}\right)^{i}=\left(w_{g}^{i}-\lambda {\frac {w_{g}^{i}}{\left\|w_{g}\right\|_{g}}}\right)\mathbf {1} _{\|w_{g}\|_{g}\geq \lambda }.}</annotation>

</semantics>

</math>

where <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \mathbf {1} _{\|w_{g}\|_{g}\geq \lambda }}">

<msub>

</mrow>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \mathbf {1} _{\|w_{g}\|_{g}\geq \lambda }}</annotation>

</semantics>

</math><img src="./12c8b2629cdb0eec08d55cf025df4d19c9ce4d59.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.505ex; width:8.236ex; height:3.343ex;" alt="{\displaystyle \mathbf {1} _{\|w_{g}\|_{g}\geq \lambda }}" loading="lazy"> is the indicator function for group norms <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \geq \lambda }">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \geq \lambda }</annotation>

</semantics>

</math><img src="./0f3436633c3d877dc7b1feacb69714cccb61a72d.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.505ex; width:3.809ex; height:2.343ex;" alt="{\displaystyle \geq \lambda }" loading="lazy">.

Thus, using <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \ell _{2,1}}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \ell _{2,1}}</annotation>

</semantics>

</math><img src="./066787a72536917c808f3d2c364921b4227e50e3.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.005ex; width:3.303ex; height:2.843ex;" alt="{\displaystyle \ell _{2,1}}" loading="lazy"> norms it is straightforward to enforce structure in the sparsity of a matrix either row-wise, column-wise, or in arbitrary blocks. By enforcing group norms on blocks in multivariate or multi-task regression, for example, it is possible to find groups of input and output variables, such that defined subsets of output variables (columns in the matrix <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle Y}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle Y}</annotation>

</semantics>

</math><img src="./961d67d6b454b4df2301ac571808a3538b3a6d3f.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.171ex; width:1.773ex; height:2.009ex;" alt="{\displaystyle Y}" loading="lazy">) will depend on the same sparse set of input variables.

<div class="mw-heading mw-heading2"><h2 id="Multiple_kernel_selection">Multiple kernel selection</h2></div>

The ideas of structured sparsity and <a href="Feature_selection" title="Feature selection">feature selection</a> can be extended to the nonparametric case of <a href="Multiple_kernel_learning" title="Multiple kernel learning">multiple kernel learning</a>.<a href="#cite_note-9">[9]</a> This can be useful when there are multiple types of input data (color and texture, for example) with different appropriate kernels for each, or when the appropriate kernel is unknown. If there are two kernels, for example, with feature maps <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle A}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle A}</annotation>

</semantics>

</math><img src="./7daff47fa58cdfd29dc333def748ff5fa4c923e3.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.743ex; height:2.176ex;" alt="{\displaystyle A}" loading="lazy"> and <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle B}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle B}</annotation>

</semantics>

</math><img src="./47136aad860d145f75f3eed3022df827cee94d7a.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.764ex; height:2.176ex;" alt="{\displaystyle B}" loading="lazy"> that lie in corresponding <a href="Reproducing_kernel_Hilbert_space" title="Reproducing kernel Hilbert space">reproducing kernel Hilbert spaces</a> <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle {\mathcal {H_{A}}},{\mathcal {H_{B}}}}">

<msub>

</mrow>

</msub>

</mrow>

<msub>

</mrow>

</msub>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\mathcal {H_{A}}},{\mathcal {H_{B}}}}</annotation>

</semantics>

</math><img src="./cd8c202368b80b37f3e644beee5e073958a0d477.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:7.863ex; height:2.509ex;" alt="{\displaystyle {\mathcal {H_{A}}},{\mathcal {H_{B}}}}" loading="lazy">, then a larger space, <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle {\mathcal {H_{D}}}}">

<msub>

</mrow>

</msub>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\mathcal {H_{D}}}}</annotation>

</semantics>

</math><img src="./dfecdce7df93f95483cb6325758cc65d48c78424.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:3.463ex; height:2.509ex;" alt="{\displaystyle {\mathcal {H_{D}}}}" loading="lazy">, can be created as the sum of two spaces:

<msub>

</mrow>

</msub>

</mrow>

<msup>

</msup>

<msub>

</mrow>

</msub>

</mrow>

<msup>

</msup>

<msub>

</mrow>

</msub>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\mathcal {H_{D}}}:f=h+h';h\in {\mathcal {H_{A}}},h'\in {\mathcal {H_{B}}}}</annotation>

</semantics>

</math>

assuming linear independence in <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle A}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle A}</annotation>

</semantics>

</math><img src="./7daff47fa58cdfd29dc333def748ff5fa4c923e3.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.743ex; height:2.176ex;" alt="{\displaystyle A}" loading="lazy"> and <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle B}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle B}</annotation>

</semantics>

</math><img src="./47136aad860d145f75f3eed3022df827cee94d7a.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.764ex; height:2.176ex;" alt="{\displaystyle B}" loading="lazy">. In this case the <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \ell _{2,1}}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \ell _{2,1}}</annotation>

</semantics>

</math><img src="./066787a72536917c808f3d2c364921b4227e50e3.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.005ex; width:3.303ex; height:2.843ex;" alt="{\displaystyle \ell _{2,1}}" loading="lazy">-norm is again the sum of norms:

<msub>

<mrow>

</mrow>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</msub>

<msub>

<mrow>

</mrow>

<msub>

</mrow>

</msub>

</mrow>

</msub>

<msub>

<mrow>

<msup>

</msup>

</mrow>

<msub>

</mrow>

</msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \left\|f\right\|_{{\mathcal {H_{D}}},1}=\left\|h\right\|_{\mathcal {H_{A}}}+\left\|h'\right\|_{\mathcal {H_{B}}}}</annotation>

</semantics>

</math>

Thus, by choosing a matrix regularization function as this type of norm, it is possible to find a solution that is sparse in terms of which kernels are used, but dense in the coefficient of each used kernel. Multiple kernel learning can also be used as a form of nonlinear variable selection, or as a model aggregation technique (e.g. by taking the sum of squared norms and relaxing sparsity constraints). For example, each kernel can be taken to be the Gaussian kernel with a different width.

<ul><li><a href="Regularization_(mathematics)" title="Regularization (mathematics)">Regularization (mathematics)</a></li></ul>

<div class="mw-heading mw-heading2"><h2 id="References">References</h2></div>

/* start https://en.wikipedia.org/ */

.mw-parser-output .reflist{margin-bottom:0.5em;list-style-type:decimal}@media screen{.mw-parser-output .reflist{font-size:90%}}.mw-parser-output .reflist .references{font-size:100%;margin-bottom:0;list-style-type:inherit}.mw-parser-output .reflist-columns-2{column-width:30em}.mw-parser-output .reflist-columns-3{column-width:25em}.mw-parser-output .reflist-columns{margin-top:0.3em}.mw-parser-output .reflist-columns ol{margin-top:0}.mw-parser-output .reflist-columns li{page-break-inside:avoid;break-inside:avoid-column}.mw-parser-output .reflist-upper-alpha{list-style-type:upper-alpha}.mw-parser-output .reflist-upper-roman{list-style-type:upper-roman}.mw-parser-output .reflist-lower-alpha{list-style-type:lower-alpha}.mw-parser-output .reflist-lower-greek{list-style-type:lower-greek}.mw-parser-output .reflist-lower-roman{list-style-type:lower-roman}

/* end https://en.wikipedia.org/ */

</style><div class="reflist">

/* start https://en.wikipedia.org/ */

.mw-parser-output cite.citation{font-style:inherit;word-wrap:break-word}.mw-parser-output .citation q{quotes:"\"""\"""'""'"}.mw-parser-output .citation:target{background-color:rgba(0,127,255,0.133)}.mw-parser-output .id-lock-free.id-lock-free a{background:url("./mw/Lock-green.svg")right 0.1em center/9px no-repeat}.mw-parser-output .id-lock-limited.id-lock-limited a,.mw-parser-output .id-lock-registration.id-lock-registration a{background:url("./mw/Lock-gray-alt-2.svg")right 0.1em center/9px no-repeat}.mw-parser-output .id-lock-subscription.id-lock-subscription a{background:url("./mw/Lock-red-alt-2.svg")right 0.1em center/9px no-repeat}.mw-parser-output .cs1-ws-icon a{background:url("./mw/Wikisource-logo.svg")right 0.1em center/12px no-repeat}body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .id-lock-free a,body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .id-lock-limited a,body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .id-lock-registration a,body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .id-lock-subscription a,body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .cs1-ws-icon a{background-size:contain;padding:0 1em 0 0}.mw-parser-output .cs1-code{color:inherit;background:inherit;border:none;padding:inherit}.mw-parser-output .cs1-hidden-error{display:none;color:var(--color-error,#d33)}.mw-parser-output .cs1-visible-error{color:var(--color-error,#d33)}.mw-parser-output .cs1-maint{display:none;color:#085;margin-left:0.3em}.mw-parser-output .cs1-kern-left{padding-left:0.2em}.mw-parser-output .cs1-kern-right{padding-right:0.2em}.mw-parser-output .citation .mw-selflink{font-weight:inherit}@media screen{.mw-parser-output .cs1-format{font-size:95%}html.skin-theme-clientpref-night .mw-parser-output .cs1-maint{color:#18911f}}@media screen and (prefers-color-scheme:dark){html.skin-theme-clientpref-os .mw-parser-output .cs1-maint{color:#18911f}}

/* end https://en.wikipedia.org/ */

</style><cite id="CITEREFRosascoPoggio2014" class="citation book cs1">Rosasco, Lorenzo; <a href="Tomaso_Poggio" title="Tomaso Poggio">Poggio, Tomaso</a> (December 2014). "A Regularization Tour of Machine Learning". <a rel="nofollow" class="external text" href="https://www.mit.edu/~9.520/fall15/Classes/learning_problem.html">MIT-9.520 Lectures Notes</a> (Manuscript).</cite>

</li>

<li id="cite_note-Candès,_Emmanuel_J_2009_pp._717-2">^ <a href="#cite_ref-Candès,_Emmanuel_J_2009_pp._717_2-0">a</a> <a href="#cite_ref-Candès,_Emmanuel_J_2009_pp._717_2-1">b</a> <cite id="CITEREFCandèsRecht2009" class="citation journal cs1"><a href="Emmanuel_Cand%C3%A8s" title="Emmanuel Candès">Candès, Emmanuel J.</a>; Recht, Benjamin (2009). <a rel="nofollow" class="external text" href="https://doi.org/10.1007%2Fs10208-009-9045-5">"Exact Matrix Completion via Convex Optimization"</a>. Foundations of Computational Mathematics. 9 (6): 717–772. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1007%2Fs10208-009-9045-5">10.1007/s10208-009-9045-5</a>.</cite>

</li>

<li id="cite_note-3"><a href="#cite_ref-3">^</a> <cite id="CITEREFZhangYeung2012" class="citation journal cs1">Zhang; Yeung (2012). "A Convex Formulation for Learning Task Relationships in Multi-Task Learning". Proceedings of the Twenty-Sixth Conference on Uncertainty in Artificial Intelligence (UAI2010). <a href="ArXiv_(identifier)" class="mw-redirect" title="ArXiv (identifier)">arXiv</a>:<a rel="nofollow" class="external text" href="https://arxiv.org/abs/1203.3536">1203.3536</a>. <a href="Bibcode_(identifier)" class="mw-redirect" title="Bibcode (identifier)">Bibcode</a>:<a rel="nofollow" class="external text" href="https://ui.adsabs.harvard.edu/abs/2012arXiv1203.3536Z">2012arXiv1203.3536Z</a>.</cite>

</li>

<li id="cite_note-4"><a href="#cite_ref-4">^</a> <cite id="CITEREFIzenman1975" class="citation journal cs1">Izenman, Alan J. (1975). <a rel="nofollow" class="external text" href="https://doi.org/10.1016%2F0047-259X%2875%2990042-1">"Reduced Rank Regression for the Multivariate Linear Model"</a>. <a href="Journal_of_Multivariate_Analysis" title="Journal of Multivariate Analysis">Journal of Multivariate Analysis</a>. 5 (2): 248–264. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1016%2F0047-259X%2875%2990042-1">10.1016/0047-259X(75)90042-1</a>.</cite>

</li>

<li id="cite_note-5"><a href="#cite_ref-5">^</a> <cite id="CITEREFKakadeShalev-ShwartzTewari2012" class="citation journal cs1">Kakade; Shalev-Shwartz; Tewari (2012). <a rel="nofollow" class="external text" href="http://www.jmlr.org/papers/v13/kakade12a.html">"Regularization Techniques for Learning with Matrices"</a>. Journal of Machine Learning Research. 13: 1865–1890.</cite>

</li>

<li id="cite_note-6"><a href="#cite_ref-6">^</a> <cite id="CITEREFArgyriouEvgeniouPontil2008" class="citation journal cs1">Argyriou, A.; Evgeniou, T.; Pontil, M. (2008). <a rel="nofollow" class="external text" href="https://doi.org/10.1007%2Fs10994-007-5040-8">"Convex multi-task feature learning"</a>. <a href="Machine_Learning_(journal)" title="Machine Learning (journal)">Machine Learning</a>. 73 (3): 243–272. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1007%2Fs10994-007-5040-8">10.1007/s10994-007-5040-8</a>.</cite>

</li>

<li id="cite_note-7"><a href="#cite_ref-7">^</a> <cite id="CITEREFHuangZhangMetaxas2011" class="citation journal cs1">Huang; Zhang; Metaxas (2011). <a rel="nofollow" class="external text" href="http://www.jmlr.org/papers/v12/huang11b.html">"Learning with Structured Sparsity"</a>. Journal of Machine Learning Research. 12: 3371–3412.</cite>

</li>

<li id="cite_note-8"><a href="#cite_ref-8">^</a> <cite id="CITEREFChenLinKimCarbonell2012" class="citation journal cs1">Chen, Xi; et al. (2012). <a rel="nofollow" class="external text" href="https://doi.org/10.1214%2F11-AOAS514">"Smoothing Proximal Gradient Method for General Structured Sparse Regression"</a>. Annals of Applied Statistics. 6 (2): 719–752. <a href="ArXiv_(identifier)" class="mw-redirect" title="ArXiv (identifier)">arXiv</a>:<a rel="nofollow" class="external text" href="https://arxiv.org/abs/1005.4717">1005.4717</a>. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1214%2F11-AOAS514">10.1214/11-AOAS514</a>.</cite>

</li>

<li id="cite_note-9"><a href="#cite_ref-9">^</a> <cite id="CITEREFSonnenburgRatschSchaferScholkopf2006" class="citation journal cs1">Sonnenburg; Ratsch; Schafer; Scholkopf (2006). <a rel="nofollow" class="external text" href="http://www.jmlr.org/papers/v7/sonnenburg06a.html">"Large Scale Multiple Kernel Learning"</a>. Journal of Machine Learning Research. 7: 1531–1565.</cite>

</li>

</ol></div></div></div><div><div class="zim-footer">

This article is issued from <a class="external text" title="Last edited on 2025-04-14" href="https://en.wikipedia.org/wiki/?title=Matrix_regularization&oldid=1285634709">Wikipedia</a>. The text is available under <a class="external text" href="https://creativecommons.org/licenses/by-sa/4.0/deed.en">Creative Commons Attribution-Share Alike 4.0</a> unless otherwise noted. Additional terms may apply for the media files.

</div>

</div></div>

</div>

</main>

</div>

</body></html>